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Abstract 

The purpose of this study was to ascertain the effects of two 
factors, the data base used for instructional decision making and the 
structure of instruction, on student achievement. The dependent data 
were the number of words read correctly by 117 students on three basal 
reading passages. Students were rank ordered in two ways: first by 
the degree of implementation o^f a technically adequate curriculum- 
based measurement and evaluation system, and second by the degree of 
structure in their instructional programs. The top 11% were compared 
to the bottom 27% for both variables. The t tests on the* mean z 
scores indicated no difference in achievement due to the , structure of 
instruction. A significant difference was found between high and low 
implementation groups on one reading passage; differences between 
scores onSlwo other passages were not significant but were in the same 
direction, supporting the hypothesis that a high degree of 
implementation of a technically adequate data system does lead to 
greater achievement. The discussion addresses implications for 
teaching practices. 
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The Impact of the Structure of Instruction 
and the Use of Technically Adequate Instructional Data 
on Reading Improvement 

The basic teaching skill is decision making (Shavelson & Borko, 
1979). "Any teaching act is a result of a decision, either conscious 
or unconscious 11 (Shavelson, 1973, p. 144). Decisions are made 
spontaneously as a teacher responds to the immediate demands of a 
situation and decisions are made in a more conscious manner as a 
teacher plans and prepares for lessons. The former decisions are 
interactive and the latter are preactive (Jackson, 1968). 

Preactive decision making allows teachers to consider a wealth of 
information, including subjective and objective data. Teachers may 
consider their own informal observation, anecdotal reports of other 
teachers, notes from parents, school records, or standardized test 
scores. Given this "information overload," teachers must sifc through 
all these data to determine which sources provide the most useful 
information and then make their planning decisions accordingly. 
Little information is available about the process teachers use to make 
these decisions or the quality of these decisions. The studies that 
address these issues are discussed below. 
The Process of Teacher Decision Making 

When making decisions concerning the placement of students in 
reading groups, teachers primarily consider student characteristics, 
which they transform into estimates of students 1 reading abilities 
(Shavelson & Borko, 1979). After reading groups are formed, the group 
becomes the focus of instruction so that the teacher f s decisions 
concerning how to teach reading are not based on individual students 
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but on the group. Basically, then, teachers summarize a great deal of 
datn into one salient piece of information, the group's ability. 
Based on this global assessment, the teacher decides how to instruct. 

Potter and Mirkin (1982) summarized responses to a questionnaire 
returned by 128 teachers of learning disabled students on how teachers 
plan and implement instruction. The analysis revealed that teachers 
most often rely on informal assessment and personal observation as the 
information used in making planning decisions. However,. Potter and 
Mirkin noted that there was great variability in the sources of 
information teachers used. These sources ranged from scores on IQ 
tests, achievement tests, and criterion-referenced tests, to parental 
input, input of other teachers, and behavioral observation. 

A major problem with the myriad of data sources used by teachers 
is their inconsistent technical adequacy. Many of the pieces of 
information on which a teacher bases decisions may be unreliable 
and/or invalid. Use of data with such flaws may Tead to erroneous 
decision making. Yet, teachers appear to be sensitive to the issue of 
technical adequacy of data, as was shown by Shavelson, Caldwell, and 
Izu (1977). They asked teachers to estimate the progress that would 
be made by a fictitious student. Some teachers received technically 
adequate information, such as test scores, and others received 
unreliable and potentially invalid information, such as a statement 
from a classmate. Teachers who received unreliable information 
revised their estimates of probability when they were given additional 
data. Teachers appeared to prefer making decisions based on 
technically adequate sources of information. 



However, as Shave 1 son and Borko (1979) and Potter and Mirk in 
(1982) pointed out, teachers use a number of technically inadequate 
sources. The technical adequacy of the information on which teachers 
base their decisions is crucial if teachers are to make accurate 
preactive decisions. And if decision making, is the basic teaching 
skill, then providing teachers with more technically adequate data 
with which to make decisions may help teachers improve the instruction 
they provide. If teachers are trained to routinely and frequently 
gather technically adequate data and use these data to make decisions 
about when to change their instructional procedures, then perhaps the 
decision-making process will be improved. 
The Quality of X&acher Decision Making 

The quality of educational decisions is best assessed via the 
primary results "of those decisions, namely, student achievement. 
Research regarding educational practices that positively effect 
student achievement suggests that structure pf instruction is an 
important factor (Stevens & Rosenshine, 1981). Structure of 
instruction is comprised of factors such as academic engaged time 
(Borg, 1980; Fisher, Berliner, Filby, Marliave, Cahen, Si Dishaw, 
1980), teacher-directed learning, pacing (Fisher et al., 1980), 
positive consequences, frequency of correct responses, correction, 
procedures, and direct practice on the target behaviors (Borg, 1980; 
Starlin, 1979). 

Research concerning the degree of structure of instruction in 
classrooms is sparse. Baker, Herman, and Yeh (1981) reported 
widespread use of games, puzzles, and adjunct devices, indicative of 



low structure of Instruction. Leinhardt, Zigmond, and Cooley (1981) 
concluded that teachers made poor Instructional decisions. This 
conclusion was reached after a year long project in which learning 
disabled students were observed in the classroom in order to ascertain 
the kind of instruction they were receiving- The structure of 
instruction for any given student may be dependent on the curriculum 
materials used. As McNair and Joyce (1979) have indicated, mos^t 
classroom activities are derived from the curriculum materials. 
Purpose 

In summary, the data teachers typically use to make instructional 
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decisions are often technically inadequate and some teachers select 
less than adequate instructional procedures with regard to 
instruction. Therefore, it would be interesting to ascertain what the 
impact would be on student achievement when both of these factors, the 
data base and structure, are manipulated. The purpose of this 
research was to examine these two variables. The specif ic-xresearch 
questions were: 

(1) What is the effect bf teacher's use of technically ^ 
adequate, curriculum-based repeated measurement and 
evaluation procedures on students' reading achievement? 

(2) What is the effect of the degree of structure of 
students' instructional, programs on students' reading 
achievement? 



n Method 
Subjects • 

A total of 31 teachers, participated 1n this study. In this 
group-, there were 26 females and £ males. On the average, they had 

1.9 years of experience teaching regular education and 8.8 years 

► . • * 

teaching special education. The -greatest percentage of teachers (39%) 
had no experience teaching regular education;' 23% had taught special 
education for one to three years. . ^ 

There were 117 students Included in the study. -Their ages ranged 
from 6 to 13 years, with -an average age of 9.5. There were 92 males 
and 23 females (the sex of two .subjects was uncoded) in grades 1-7. 
The greatest numbers of students were in grades 2-5 (20, 26, 25, and 
25, respectively). In grade 1, there were five students, in.grac[e-6, 
there were nine students, and in grade. 7, there were two students. 
The students included in the study were, for the most part (111 of the 
117),' provided with special education in resource rooms. 
Measures 0 

- % 

p 

Three major types of measures "were employed in this study. 
First, a measure of the degree of implementation of the measurement 
and evaluation procedures was included since it was critical to know 
how accurate and complete teachers were in using the evaluation 
system. Second, measures indicating the degree of structure of the 
students' instructional programs were included. These measures were 
useful in determining how the evaluation system influences teaching 
practices. Both the implementation and structure measures served as 
independent variables. The third set of measures included student 
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achievement indices, which served 'as the dependent data. 

Implementation variables . The Accuracy of Implementation Rating 

Scale (AIRS) 1s an instrument that was developed in conjunction with 
— \ . *■ 

the manual Procedures to Develop and Monitor Progress on IEP Goals 
(Mirkin, Deno, Fuchs, Wesson, Tindal, Marston, & Kuehnle, 1901), which 
was used for teacher training in this study. The AIRS 'provides a 
format for monitoring , the implementation of the procedures described 
in the manual. The AIRS consists of 12 items rated on a 1 to 5 scale, 
1 being the lowest implementation score and 5 being complete and 
accurate implementation. A complete list of the items and their 
operational definitions can be found in Appendix, A. 

Items 1 and. 2 of the AIRS, which require direct observation, deal 
with the accuracy of administration of the measurement and felection 
of the stimulus materials. - Items 3-12 of the AIRS require inspection 
of various written documents. Specifically, the rater examines four 
documents for each student: the Individualized Educational Plan (IEP), 
which should specify the long-range goal and short-term objective in 
reading, the reading graph, the instructional plan for reading, and 
the record of changes made in the instructional plan in reading. 
Factors included in items 3-12 pertain to the establishment of (a) the 
appropriate measurement level; (b) an adequate baseline; (c) an 
accurate long-range goal and short-term objective; (d) a detailed 

J 
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graph; (e) a complete instructional program; (f) a correct aimline; 
(g) the timing of' instructional changes; (h) the clarity and intensity 
of the changes made. (See Appendix A.) The AIRS was used to assess 
the degree of implementation at the beginning, mid-way, and at the 



conclusion of this study. 

The Interjudge agreement for the AIRS ranged from .92 to .98 when 
percentage of agreement was based on a w1th1n-one-po1nt rating match. 
The percentage of exact agreement ranged from .73 to .91. 

Structure variables . The Structure of Instruction Rating Scale 
(SIRS) 1s designed to measure, through observation, the degree of 
structure of the Instructional lesson that a student receives.' In 
this study, the focus was on structure during reading Instruction. 
The variables chosen for Inclusion on the SIRS were gathered from 
current literature on Instruction and student academic- achievement 
(cf. Stevens & Rosenshlne, 1981). A 11st of the variables and their 
operational definitions can be found in Appendix B. Observations were 
conducted at three different points 1n time during the study'.' 

The SIRS consists of 12 five-point rating scales 1n which a rating 
of 1 1s. low for the variable and 5 1s high. Observers, trained by 
videotape to a criterion of .80-. 90 inter-rater agreement, rate all 
variables on the basis of , strict definitions at the end of a 20-m1nute 
observation period. For the present study, nine research assistants 
were trained as observers; they reached an Inter-rater agreement level 
of .92 before actually observing in classrooms. The focus of each 
observation period for the SIRS is on the Instructional environment 



Alpha, a measure of Internal consistency. For a sample of 70 students 
observed in November 1981, the average inter-item correlation was .37, 
resulting in an alpha of .86. Thus, the SIRS seems to have a high 
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degree of reliability as indexed by a homogeneity measure. 

** Achievement measures . Three one-minute oral reading measures, 

consisting of randomly selected passages from the third grade level in 

Wwi 720, were administered to. the students at the end of the study. 

These measures were selected based on their technical adequacy (Deno, 

Mirkin, & Chiang, 1982) and sensitivity to\change (Marston, Lowry, 

Deno & Mirkin, 1981). These simple measureY are as reliable and 

• <• v ■ 

valid is traditional standardized tests and yet are more likely to 

reflect small increments i n improvement. The measurements were 

conducted by directing students to begin reading at the, top of the 

page and continue reading for one minute, at which tinie the examiner 

would say stop. If they came to a word they did /lot know, the 

examiner -would supply the word arid prompt "them to- continue. While, the 

student was reading, the examiner followed along on a copy * of the 

passage and - marked errors of -substitution and omission. Following the 

;reading,th> numbers of words read correct and incorrect were counted 

and recorded, with no feedback given to the student. These three 

reading measures were given at the beginning of the study and 

immediately following the final observation (posttes^t). A gain score 

then was' calculated for each subject; 

: ' ' * V V"".' .;' ■ . 

Procedures . ' - " • ; ' •>••->.• 

All teachers Were trained .to carry out a specific set of 
procedures, including establishing an appropriate measurement level, 
writing* long-range goals (LRGs) and short-term objectives (STOs), 
collectings three oraf reading scores per week for each student, 
plotting the scores on a graph, and using the data in making decisions 



about the effectiveness of students 1 instructional programs. 

Measurement . Reading measurement consisted of one-minute timed 
samples of reading from the student's curriculum. Both words correct" 
and incorrect were scored and charted on equal interval charts. The 
level of stimulus material for testing, which also became the 
baseline, was selected as the* level from which the student could read 
aloud between 20-29 words per minute for grades 1 and 2, and 30-39 
words per minute for grades 3-6'. 

Writing goals . Teachers were instructed to write long-range 
goals for the student's IEP using both the entry level criterion and a 
desired year-end mastery criterion, usually 70 words correct per 
minute with no more than 7 errors. The format used in writing the 
long range goal is shown in Figure 1. 



Insert Figure 1 about here 



Writing objectives . Two types of short-term objectives were 
written, performance and mastery; both were based on the long-range 
goals. For performance objectives, in order to compute the short-term 
objective, teachers first subtracted the baseline level of performance 
from the criterion level listed in the LRG.' Dividing this difference 
by the number of weeks necessary until the annual review, they arrived 
at the number of words per week gain necessary to meet the long-range 
goal criteria. In performance measurement, the measurement task is a 
random sample. of items from a constant set of stimuli, and the goal is 
to improve the level of performance on that stimulus material. In 
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graphing performance measurement, the horizontal axis represents 
successive school days and the vertical axis represents the level of 
performance on a constant measurement task; each data point represents 
the level of proficiency on that constant measurement task. The line 
of best fit through the data points depicts the student's rate of 
improvement in performance on the set of stimulus material. 

When writing mastery based short-term objectives, teachers 
backtrack through the reading curriculum to find the level at which 
the student reads at the mastery rate designated in the long-range 
goal. The pages or stories between this baseline level and the goal 
level are counted and divided by the number of weeks until the annual 
review. This number becomes the criterion used in the STO specifying 
the average weekly progress necessary to meet the i-RG. On the graph, 
the horizontal axis again represents school days and the vertical axis 
represents successive segments, pages, or stories of the curriculum 
mastered. Each data point represents the number of curriculum 
segments mastered through a given day. The line of "best fit through 
the data points depicts the rate of student progress through the 
curriculum. The purpose, of repeated mastery assessment is to assess 
the student's rate of mastery in the curriculum, and the purpose of 
the graph is to display that rate of curriculum mastery. The teacher 
measures the student on a representative sample of material from the 
current instructional curriculum unit and plots that level on the 
graph until mastery is achieved-. At that point (a) the teacher 
registers on the student's graph that a curriculum unit has been 
mastered, and (b) the set of reading stimulus material on which the 

15 
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teacher measures the student progresses to the next segment in the 
hierarchy. The two formats used for writing short-term^ objectives are 
listed in Figure 2. , 

Insert Figure 2 about here 



Data utilization . In addition to measuring and writing goals and 
objectives, the teachers were trained in the use of the measurement 
procedures for evaluation of the instructional program. In order to 
monitor student growth, the baseline reading level and the long-range 
goal were c: :ted by an aimline that showed the student's desired 
progress. .Every seven data points, the teachers were to monitor 
• student growth by means of the split-middle or quarter-intersect 
method (White & Haring, 1976). An example is given in Figure 3. If 
the student was progressing at a rate equivalent to or greater than 
that indicated by the aimline, the instructional program was 
continued;, if the projected rate of growth was less than that 
indicated by the "a'imline, teachers were directed to make a major 
change in the student's instructional program. 



Insert Figure 3 about here 



Teacher Training 

Three formats were used to train teachers /in these procedures. 
For 10 teachers in one special education cooperative, training in the 
use of the measurement procedures took place in a series of three 

\ . , 
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half-day workshops at ifheleg inning of the school year. Teachers also 
were provided with the manual, Procedures to Develo p and Monitor 
Progress on IEP Goals (Mirkin et al., 1981), which detailed all the 
Activities teachers were to do. In addition, visits by observers in 
December, February, and May, and frequent phone contacts, provided 
feedback to" the teachers on the accuracy of their implementation of 
the measures: 

In two other districts, training was conducted by district 
personnel with the aid of the same manual. In November, individuals 
designated by each district as trainers participated in a one-day 
trainer's workshop. At this time the procedures were reviewed for the 
trainers and they were , given trainer's manuals that specified 
activities for them to use when teaching the monitoring procedures to 
the teachers. After this trainer's workshop, the trainers set up and 
conducted a series of training sessions in their own districts. 
Ques/tions about . the procedures usually were forwarded to IRLD 
personnel. On-going phone contact facilitated the training process. 

" The last type of teacher training involved 10 teachers from a 
rural special education cooperative that had served as a pilot site. 
These teachers were trained during one week of full-day workshops 
prior to the 1980-81 school year and during monthly, half-day 
workshops throughout the year. These workshops were conducted by IRLD 
staff and, prior to February, their focus was on training the teachers 
to (a) write curriculum-based IEPs, (b) create a curriculum-based 
measurement procedure including mastery and performance systems, (c) 
measure frequently and graph, student progress toward IEP goals, and 

17 
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(d) develop strategies to improve the feasibility of implementing the 
frequent measurement systems. By February, each teacher had 'developed 
curriculum-based lEPs for at least two students and was measuring and 
graphing those students 1 reading performance at least three times each 
week. In February, the data-utilization systems were introduced to 
the teachers. The remainder of the workshops consisted of teacher 
presentations of their graphs and discussions of student progress and 
changes in instructional p,ans. ^ 
Data Collection 

Thrdughout the year, specific data were compiled by each teacher 
and sent to an IRLD staff member who was designated as the contact 
person. Data collection took place on three occasions, separated by 
approximately two months each, and was synchronized with the SIRS and 
AIRS observations. 

Each teacher compiled a packet for each student in the study. 
The packet consisted of the following forms: (a) SIRS; (b) AIRS; (c) 
Graph; (d) IEP (IRLD form); (e) Instructional . Plan (IRLD form); (f) 
Changes in Instructional Plan (IRLD form); (g) Student Information 
Sheet; and (h) 3rd Grade Passage Scores. 

To insure confidentiality, each student was assigned an ID number 
and names were removed before the documents left the districts. The 
information obtained from the teachers was glearted by research 
assistants according to the implementation, structure, and achievement 
variables. 
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Observer Training 

In order to collect SIRS data and rate items 1 and 2 on the AIRS, 
observations of each student during reading class were necessary. 
Staff members (lead teachers, program coordinators) from two locations 
involved, in the research carried out the necessary observation 
procedures in their districts. These observers were trained during 
one half-day session by two IRLD staff members. A brief review of the 
research design was provided at the onset of the training. The 
primary focus of the training was on actual observation procedures 
required of the observers throughout the -year, particularly proper use 
of the Structure of Instruction Rating Scale (SIRS) and the Accuracy 
of Implementation Rating Scale (AIRS). 

Explanation of the SIRS included its history and rationale, its 
purpose, and its administration procedures. Each item on the scale 
was discussed in detail, including definitions for and examples, of 
several ratings per item. After the SIRS was explained, two 
videotapes were used as a training aid to gi ve the-\observers a chance 
to practice their skills. The tapes consisted of two resource room 
situations, one demonstrating a model teacher and the other more 
indicative of a teacher who would receive lower ratings on the SIRS. 
Each item on both tapes was rated by each observer and an IRLD staff 
member and discussed. An inter-rater agreement of .80 was required of 
the observers before the session ended. 

' The AIRS training consisted of explanations of the two items oh 
the scale that the observers would be rating. The final portion of 
the training involved the organizational aspect of the data 
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collection. A list of documents that were to be collected at the time 
of each observation was drawn up and explained. Throughout the year, 
an IRLD staff member was in contact with the observers ;on a weekly 
basis to insure understanding and consistency of the procedures and to 
answer any questions,, 

In the other two study sites, trained IRLD staff members 
conducted the observations. Nine observers were used in one district 
and four in the other. Training of these observers was similar to the 
training of the district personnel. The videotape and code book were 
presented and ratings were practiced until the required level of 
interobserver agreement was reached. 
Data Analysis . . 

Data analysis was done via a series of steps. The first step was 
to standardize the reading passage scores by grade level in order to 
minimize the influence of grade on achievement. Second, the three 
implementation and three structure scores were ; averaged across the 
three data collections to arrive at an average implementation and an 
average structure score per subject. The third step was to 
standardize the structure and implementation scores by school district 
in order ' to minimize the measurement error introduced by using 
different raters at the various sites. The fourth step in the data 
analysis was to compile rank order distributions of the 117 students; 
one rank order distribution was according a to standardized average 
implementation score, and the second rank order distribution was 
according to standardized average structure score. 

The upper 27% and lower 27% of each distribution composed the 
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groups used for this analysis. ' The gain scores of the high 
implementation group were compared to the gain scores of the low 
implementation group and the gain scores of the high structure group 
were compared tothe gain scores of the low structure group. J tests, 
were employed for these comparisons; the dependent data were the gain 
scores from pretest to posttest. 

Results 

Implementation 

Table 1 presents the mean z score gains for Low and High 
implementation Groups, as well as the results of the t_ tests. The 
matched pairs t, tests on the gain scores from the . three passages 
yielded a significant difference between groups (£ = .004) for passage 
1: the High Implementation group, showed greater gains in achievement 
than the Low Implementation group. The sample mean gains also were 
higher for the High Implementation group on passages 2 and 3; however, 
the t_ tests on the gain scores for these two passages were not 
significant. 



Insert Table 1 about here 



-Structure 

^Table 2 presents the gain scores for Low and High Structure 
Groups^asxwell as the results of the t_ tests. None of the t_ tests 
between achievement gains of the High and Low Structure groups yielded 
significant resuVtsV In addition, no trend was apparent in the data 
to indicate that eitheXthe Low or High Structure group showed more 



improvement than the other group. 



Insert Table 2 about here 



Discussion 



The results lend modest support to the conclusion that teachers 1 
use of a technically adequate, curriculum-based repeated measurement 
and evaluation system positively affects their instructional 
decisions. In this study the students of those teachers who more 
accurately and consistently applied this system made better progress. 
Apparently, the teachers made better instructional decisions using the 
technically adequate data and consistent rules regarding how to use 
that data. 

In contrast, the structure of instruction, as measured in the 
study, appears to have no bearing on reading improvement. The 
students of teachers who conducted highly structured programs improved 
no more than the students of less structured teachers. This fWding 
is important because 1n a descriptive study such as this one many 
rival hypotheses can be offered for relationships obtained between 
presumably causal variables. For example, the relationship between 
high implementation of measurement and evaluation and achievement 
might, ordinarily, be explained by offering another construct- 
particularly structure— as the cause of improved achievement. In such 
an analysis high implementation would be seen as concommitant rather 
than causal. Given the present results, however, structure of 



instruction as the "real cause 11 of Improved reading performance 




18 

be offered as a plausible rival hypothesis. 

Considering the important role given to structure by other 
researchers, however, some speculation about the lack of findings for 
this variable should be offered. First, the SIRS may not be an 
adequate tool to assess structure. Perhaps other structure variables 
would be more salient than those used in this scale. The research on 
which the SIRS is based focuses on regular education. It is possible 
that those variables are not as .powerful for students in special 
education. Other aspects of instruction may be more important with 
special populations. A second possible explanation for , lack of 
effects is the sampling procedure used to collect the SIRS . ratings. 
Students were observed for 20 minutes on three occasions. Therefore, 
a total of approximately 60 minutes of classroom time was used to 
represent instruction received over a seven-month period. It is 
impossible to know whether the SIRS scores accurately .reflect the 
structure of instruction on the whole. The observers reported that 
some teachers made comments after the observations that their students 
were behaving differently than usual. Perhaps the teachers also 
instructed differently when observed. 

The main implication for practice that can be derived from this 
study is that teachers should be encouraged to use a technically 
adequate data base for making instructional decisions. The results of 
the structure of instruction analysis may N indicate that teachers 
cannot always rely on a model set of instructional procedures that 
generally seem to have a positive effect on achievement. No single 
instructional procedure will work for every student. Rather, teachers 
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must consider each student individually and regard each instructional 
procedure as a hypothesis that must be tested to determine whether it 
works for a particular student. The measurement procedures described 
in this paper meet the technical characteristics that are necessary 
for teachers to use the data with confidence. These data can help 
teachers make better instructional decisions for each student and 
these decisions result in improved performance. 



ERLC 



24 



20 * „ 

References 

Baker, E., Herman, J. L., & Yen, J-; P. Fun and games: Their, contri- 
bution to basic skills instruction in elementary school. 
American Educational Research Journal , 1981, 18, 83-92. < 

Borq, W. Time and school learning. In C. Denham & A. Lieberman 

(Eds.), Time to learn . Washington, D.C.: National Institute of 
Education, 1980. 

Deno, S. L., Mirkin, P. K., & Chiang, B. Identifying valid measures 
of reading for use in continuous evaluation of .educational- 
programs. Exceptional Children , 1982, 49, 36-47. 

Fisher, C, Berliner, D., Filby, N., Marliave, R., Cahen, L., & 

Distfew M. Teaching behaviors, academic learning time, and stu- 
defe achievement: An overview. , In C, Denham & A. Lieberman 
(Eds.), Time to learn . Washington, D.C. National Institute of 
Education, 1980. 

Jackson, P. Life in the classroom . New York: Holt, Rinehart and 
Winston, 1968. " 



Leinhardt, G. , Zigmond, N., & Cooley, W. Reading. instruction and its 
effects. American Educational Research Journal , 1981, _18, 
343-361. 

Marston, D., Lowry, L., Deno, S., & Mirkin, P. An analysis of 

learning trends in simple measures of read ing, spelling, and 
wr itten expression: A lonqitudinal study (Research Report No. 
.497: — Minneapolis: University of Minnesota, Institute for 
Research on Learning Disabilities, 1981. 

Mirkin, P., Deno, S., Fuchs, L., Wesson, C, Tindal, G., Marston, D 
tit Kuehnle, K. ProceduresH^eveijBTKand monito r progress on IEP 
goals . Minneapolis: JSniwsj^df^nesota, institute tor 
Research on Learning 




McNalr, K., & Joyce, B. Teachers 'Hhfr johts wh>K teaching: The South 
Bay st udy. Part II (Research Series No. bb). Lansing, Mich.: 
"Michigan State University, The Institute for Research on 
Teaching, 1979. 

Potter M., & Mirkin, P. Instructional planning and impl ementation 
p ractices of elementary and secondary resource room teachers: 
Is there a difference? (Research Report No. 65). Minneapolis: 
University of Minnesota, Institute for Research on Learning 
Disabilities, 1982. 

Shavelson, R. J. What 1s the basic teaching skill? The Journal of 
Teacher Education , 1973, 14, 144-151. 



Shavelson, R. J., & Borkc, H. Research on teachers 1 decisions in 
planning instruction*. Educational Horizons , 1979, 57, 183-189. 



Shavelson, R. J., Caldwell, J., & Izu, T. Teacher's sensitivity to 
the reliability of information in making decisions. American 
Educational Research Journal , 1977, 83-97. 

Starlin, C. M. Evaluating and teaching reading to "regular" kids. 
Iowa Perspective , December, 1979. 

Stevens, R., & Rosenshine, B. Advances in research on teaching. 
Exceptional Education Quarterly , 1981, 2/1), 1-10. 

White, 0. R., & Haring, N. 6. Exceptional' teaching: A multimedia 
training package . Columbus, OH: Charles E. Merrill, 1976. 



v 



2G 



Table 1 

T-Tests on Gain. Scores for High and Low Implementation Groups 









Mean 


Standard 


Standard 








N 


z score 


Deviation 


Error 


T Value 


P Value 


Passage 1 


i 












Low Implementation 


24 


-.339 


.587 


.120 


, -3.03 


.004 


High Implementation 


18 


.309 


.801 


.189 






Passage 2 










C 




Low Implementation 


24 


-.337 


.510 ' 


.104 


-1.59 


.127 


High Implementation 


16 


.085 


.977 


.244 






Passage 3 














Low Implementation 


24 


-.561 


1 .002 


.204 


-1.52 


.138 


High Implementation 


16 


.016 


1 .407 


.352 
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Table 2 

T-Tests on Gain Scores for High and Low Structure Groups 





N 


Mean 
z score 


Standard 
Deviation 


Standard 
Error 


T Value 


P Value 


Passage 1 














Low Structure 


, 34 


.218 


.797 


.137 


1.51 


.137 


High Structure 


24 


-.087 


.732 


.146 






Passage 2 














Low Structure 


34 


.255 


.955 


.164 


-.10 


.923 


High Structure 


25 


.277 


.795 


.152 






Passage 3 














Low Structure 


34 


.366 


1 .688 


.290 


.73 


.470 


High Structure 


25 


. .100 


1 .116 


.223 
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Condition 



Behavior 



Criteria 



LRG:< 



In ■ . weeks, when 

(total i weeks) 
presented with, stories from * 

Level , 

Tlf) (reading series), 



student will 
read aloud 



at the rate, of 50 

wpm or better 

5 or fewer errors. 



Figure!. Format for Long-Range Goal : Reading 



29 



ERIC 



V 



25 



c 



o c 

E «a 
O 

&. 

0) 



CONDITION 



BEHAVIOR 



CRITERIA 



Each successive week, when 
presented with a random 

selection from_ 

(level sf from current 
instructional level - same 
as LRG) 

of . 

(reading series) 



student will 
read aloud 



at an average increase 

of " 

(70 or 50 wpm - actual 
performance) total # 
weeks remaining in 
school year. 



CONDITION 



BEHAVIOR 



CRITERIA 



Each week, when presented 
with successive stories 

from ^ 

(Level =s from current 
instructional level to 
annual goal level) 



student will 
progress 



at the rate of ' 

stories per week maintain- 
ing the mastery criteria 
of at least 50 wpm (gr . 1 
& 2) with 5 or fewer errors 
and 70 wpm (gr. 3-6; with' 
7 'or fewer errors ■ 



figure 12. Performance and Progress Charting Short Term Objectives for 
1 * Reading. 
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Figure 3. 



Using the split-middle technique to monitor student progress. 
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Appendix A 
Accuracy of Implementation Rating Scale 

School :_ . Student: - ; 

Oate: "_ Teacher ' 

Observer (Items Land 2): ; ■ ; 

Rater (Items 3-13) ; , 1 , 

Number of observations prior to rating: ' _ 

Time observation begins: Time observation ends: 

Time allocated to reading Instruction per day: , 

Curriculum used for measurement: Publishe r 

Series Level 

Instructions 

Circle the number that accurately reflects your rating for each 
variable. Only one number may be circled per variable. 1 reflects a 
low level of implementation and 5 means total implementation of the 
Procedures to Develop and Monitor Progress on IEP Goals. See Operation- 
al Definitions'. Items 1 and 2 require direct observation of the measure 
ment administration. Items 3, 4, 5, 6, and 7 require inspection of the 
student graph. 1 Items 8, 9, and 10 require inspection of the student's 
IEPform. The Instructional Plan must be Inspected to rate Item 11. 
The Change Record must be inspected to rate items 12 and 13. 







Administering the Measurement Task 


1 2 


3 


4 


5 

« — 


ii 

o o | 




Selecting the Stimulus Material 


1 2 


3 


4 


5 




~3. 


Sampling for Instructional Level 


1 2 


3 


4 


5 




4. 


Baseline ^ k 


1 2 


3 


4 


5 


0 


5. 


Graph Set-up-"^ 


1 2 


3 


4 


5 




6. 


Aimllne , 


1 2 


3 


4 


5 




7. 


Timing of Instructional Changes 


1 2 


3 


4 


5 


u 


~8. 


Long-Range Goal 


1 2 


3. 


4 


5 


o> o. 

CLUJ 


9. 


Short-Term Objective 


1 2 


3 


4 


5 


C 


ID. 


Measurement System 


1 2 


3 


4 


*5 


lisp. 

[nst. 
'Ian 


ID- 


Instructional Plan 


1 2 


3 


4 


5 


r-* u. 

n oi ' 




Substantial Changes 


1 2 


3 


4 


5 


Insi 

Chan' 
Rec 


\n: 


One, Clear Change 


12 


3 


4 


5 
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Operational Definitions 
Accuracy of Implementation Rating Scale 
Administering the Measurement Task 

5 - The measurement task is administered correctly: teacher 
brings stopwatch and pencil to measurement area; gives 
correct directions for the task; administers the measure- 
ment procedure for one minute; correctly marks the teacher 
copy; correctly counts words correct and incorrect; cor- 
rectly counts words correct and incorrect; correctly 
plots the data point. 

1 - The teacher: -forgets necessary materials; does not give 
directions; does not time the task accurately; fails to 
mark the teacher copy or incorrectly marks errors; miscounts 
correct and incorrect words; and inaccurately plots the data 
point. 

Selecting the Stimulus Material . 

5 - The teacher has followed these procedures: Uses passages 
selected from the level that represents the annual goal. 
Observers should record the book from which the passage 
was selected and later check this with the long-range goal 
level. At this level find the pages in these stories that 

do not hav e exces s i ve d ialog ue, indentati 0 ns-,- a nd/or unusual 

pronouns. Write these page numbers on equal size slips of 
paper. 

- Put the slips of paper into a drawbag and shake it. 

- Randomly pick a slip of paper. 

- The page number chosen is the page where the student 
begins reading. If the page chosen is a passage that 
was read earl ier during the week, draw another page 
number. 

Other completely random procedures are also rated a 5. If, 
however, not all passages have an equal chance of being 
selected, a 4 rating would be indicated. 

1 - The teacher fails to randomly pick the passage or the sample 
taken from a domain which is greater or smaller than the one 
indicated in the goal . 

Sampling for Instructional Level 

5 - The teacher has sampled from higher or lower reading levels 
to find the level in which the student reads 20-29 wpm 
(grades 1 &>2) or 30-39 wpm (grades 3 and up). 
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1 - The teacher is. measuring at a level which is too high or 
too low. 

4. Baseline 

5 - The student's performance has been measured at least 3 times to 
establish a stable baseline. A stable baseline means that all 
data points fall within a range of 10. 

1 - The teacher has not found a level for which a 'stable baseline 
has been established or has failed to collect 3 data points 
during the baseline phase. 

5. Graph Set-Up 

The graph is accurately set up: The dates filled in on the 
horizontal axis; the vertical axis is correctly labeled words 
read per minute from ^material; the units of measure- 
ment are specified; the student's name and subject area are 
certified; a key. identifies the symbols for correct (.) and 
incorrect (x); symbols are placed at the intersection of date/ 
and score; the data points are connected with straight lines; 
and absences are recorded on the graph as (abs.). 

1 - The graph does not include many of the items mentioned above; 

* > . 

6. Aimline 

5 - The long-range goal is marked on the graph with'an X at the 
intersection of the desired performance level and date of 
attainment and a line of desired progress connects the 
point representing the student's median score of the last 
3 data points from baseline and the LRG. 

1 - The long-range goal is not marked on the graph and/or the 
median and LRG are not connected. 

7. Timing of Instructional Changes 

5 - All the adjustments in the student's program are made at the 
appropriate time given the rules for data utilization:. 

(1) Compare the actual slope based oh 7 to 10 data points 
to the slope required to attain the Annual Goal. 

(2) If the actual slope is equal to, or steeper than, the 
Annual' Goal slope, continue the program. 

(3) If the actual slope is flatter tharuthe Annual Goal 
slope, change the program. 

1 - None of the adjustments in the student's program are made 
at the appropriate time. 
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Long-Range Goal 

5 - The long-range goal 1s accurately written; goal specifies 
the number of weeks until next review; stimulus materials 
for the goal represents the level in which the student.. 
1s performing at entry level criterion; goal specifies 
student behavior; goal specifies mastery criterion of 
50 wpm with fewer than 5 errors (grades 1 & 2) or 70 wpm 
with fewer than 7 errors (grades 3-5) when there are 35 
weeks until the annual review. If there are fewer than 35 
weeks, the criteria can be lowered proportionately . 

1 - The long-range goal contains none of the above criteria. 
Short-Term Objective 
5 



ase 




1 - The short-term objective contains none of the above criteria. 
Measurement System 

5 - The. teacher has Indicated how the material 1s organized the 
frequency of measurement, and what Is to be recorded on the 
graph. 

1 - The measurement system 1s not specified. 
Instructional Plan 

5 - The instructional plan Includes clear and specific descriptions 
of the instructional procedures, the time spent in each acti- 
vity, the pertinent materials, the arrangements, and tne 
motivational strategies. 

1 - The instructional plan is unclear and lacks specific descrip- 
tions of the Instructional procedures, the time spent in each 
activity, the pertinent materials, the arrangements, and tne 
motivational strategies. 

Substantial Changes 

5 - The adjustments In the student's program are always^substantial 
(have a good chance of being effective; see. Unit XIYJ. 

1 - The adjustments are never substantial. 
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13. Clear Change 
5 



All the adjustments made introduce only one, clear program 
change. 



1 - All the adjustments made introduce more than one change 
and/or the change is unclear. 



a \ 
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Appendix D *, 

Structure of Instruction Rating Scale (SIRS) 

School: , Student: 

Date: Teacher: , _ 



Observer: : Number of Students In Group: 

Number of observations prior to rating: _ ' 

Time observation begins: Time observation ends: . 

Time allocated to reading Instruction per day:__ , 

CurHrulum used for Instruction: Publ1sher___ . 

Series Level 

Instructions 

Circle the number that accurately reflects your rating for each 
variable. Only one number may be circled per variable. If you are 
unable to evaluate a certain variable, mark N/A (not applicable) next 
to the, left-hand column. 

1. Instructional Grouping 12 3 4 5 

2. Teacher-directed Learning 1.23 4 5 

3. Active Academic Responding 1 2 34 5 

4. Demonstration/Prompting .1 2 3 4 5 

5. Controlled Practice 12 3 4 5 

6. Frequency of Correct Answers 1 2 3 4 , 

7. Independent Practice 12 3 4 5 

8. Corrections 1 2 3 45 

9. Positive Consequences 1. 2 3 4 5 

10. Pacing 1 2.3 4 5 

11. Oral Practice on Outcome • 

Behavior 1 2 • 3 4 d 

12. Silent Practice on Outcome . 

Behavior 
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SIRS 



Operational Definitions Codebook 



Instructional Grouping 

5 - 90% or more of the Instruction this student receives from the 
teacher 1s on an Individual basis. 

1 - 10% or less of the Instruction this student receives from the 
teacher 1s on an Individual basis. 

Teacher-Directed Learning 

5 - Student's Instruction 1s extremely organized, businesslike, 
and teacher 1s firm 1n direction and control of activities. 
For example, student is presented with questions, student 
has material to cover, etc. 

1 - Student's instruction 1s casually organized and very spon- 
taneous. Teacher is .not committed to having the student work 
on a particular set of material. Instructional materials do 
not determine what activities student engages in and the les- 
sons change according to problems or mood of this student. 

Active Academic Responding 

5 - The student is actively practicing the academic skills to be 
learned more than 75% of the time observed. Specifically, the 
student is engaged in oral or written responding to teacher 
questions or written material , e.g. , reading aloud, answering 
questions, writing, or computing. Student rarely is involved 
in non-academic conversations with teacher or other students 
Attending to the lesson without responding, such as sitting, 
looking, listening, and/or following along in a book does not 
apply. The student must make an active, written or oral 
response. 

1 - The student is actively practicing the skills to be learned 
less than 10% of the time observed. Instructional lessons 
may be interrupted or shortened to include "process" and other 
non-academic activities, e.g., clarifying feelings, opinions, 
and working on arts and crafts. 

Demonstration and Prompting 

5 - Appropriate steps of the desired behavior to be performed are 
demonstrated for the student. Student is given an opportunity 
to practice the step(s) as teacher provides prompts for correct 
behavior that approximates or achives desired response. 

1 - Teacher attempts to teach the student a behavior without using 
demonstration and prompting techniques. 



.; 38 



B-3 



SIRS 

5. Controlled Practice 

5 - Student's practice of material 1s actively controlled by 
teacher who frequently asks questions to clarify that the 
, student understands what has just been demonstrated. Ques- 
tions are convergent ( single factual answer ) and the stu- 
dent's answers consistently follow the questions and are 
given teacher feedback. 

1 - Student 1s rarely questioned by teacher following demonstra- 
tion of new materials. Questions. are more divergent (open- 
ended, several Interpretations) than convergent (single factual 
answer). Student's response 1s not consistently followed by 
teacher feedback. The type of questions are such that several 
answers are acceptable, I.e., questions are abstract or am- 
biguous. 



Examples: 

If during an oral reading session: 

a) the teacher frequently attempts to clarify the material with 
convergent questions ("what color hat was John wearing?"), a 
5 would be recorded. 

b) the teacher asks few questions, most of which are divergent 
("What do you think this means?"), a 1 would be recorded. 

c) the teacher asks few convergent questions or many divergent 
• questions, the appropriate rating would be a 3. 

6 . Frequency of Correct Answers ( 

5 - Academic lessons are conducted 1n such a way that the difficulty 
of the material allows the student to achieve mean accuracy 
of BO* or higher. 

1 - Academic material Is difficult for student, component steps 
are large or unsequenced, and mean accuracy for student 1s 
less than 55%. 

(Note: If the student has no opportunity for oral or written response 
during the observational period, Item 6 would be rated N/A - 
not applicable, while Items 3 and 5 would most likely be 
rated 1.) . 

7. Independent Practice 

V ^ 5 - When engaged 1n Independent seatwork, the student frequently 1s 
monitored by the teacher who assists, clarifies, and praises 
a the student for academic engaged tasks. 

Independent seatwork 1s defined here as a student working on an 
assigned task for at least 5 minutes.., LIf no such 5-minute 
block of time 1s observed, Item 7 1s rated N/AJ.) 
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1 - When student 1s engaged 1n academic seat-work act1y1t1.es, little 
attention 1s given by teacher who directs seat-work activities 
from a distance or engages 1n work separate from the assigned 
seat work. Teacher 1s generally not helpful or supportive to 
student during Independent practice time. 

8. Corrections 

5 - The student's errors are consistently corrected by the teacher. 
When the student either does not respond, responds Incorrectly, 
or does not respond In unison 1f the activity Is group directed 
and requires such responding, the teacher will systematically 
attempt to correct the student by asking a simpler question, re- 
focusing student's attention to elicit correct response from the 
student or provide general rules by which to determine the 
correct answer 902 or more of the time | j 

1 - Student's errors are rarely and Inconslitently corrected by the 
teacher. The student responses are no,t| systematically corrected, 
Student's errors are corrected 50% or less of the time. 



For example: In oral reading this Includes teacher correction of jklps 
and mispronunciations, or hejp 1n sounding out hesitations. 



T 

9. Positive Consequences / 



5 - Positive events (tokens, points, activities, etc.) are given to 
the student when performing the desired behavior. When learning 
a new skill the student receives positive consequence for 
approximations of the desired behavior. Consequences are con- 
sistently received during academic training time. Praise and 
compliments, e.g., "good working, nice job," are not Included 
1 n this definition. 

1 - Student rarely receives positive consequences for academic work. 
When student receives, consequences they usually are for social 
behavior, rather than for behaviors occurring under systematic 
academic training. 



10. Pacing 



5 - The pace of the lesson 1s rapid, providing many opportunities 
for response by the student. As a result, attention is high 
' and off-task behavior is low. 

1 - The pace of the lesson 1s slow and the student's rate of 
responding 1s low. Lesson format, frequently varies, 1s not 
highly structured, and student attention may be low. 
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11- Oral Practice on Outcome Behavior 

5 - Student reads aloud from context nearly all the time (85-100* 
or 12-15 min. of a 15 min. observation). 

1 - Student does not read aloud during the observation (0% of the 

time). . 
(Note- Reading aloud for measurement purposes should not be considered 
* when rating this variable. Reading in context is defined as 

reading phrases, sentences, paragraphs, ..or story selections.) 

Examples: „ 

If the student is reading isolated words nearly the entire time, 
the appropriate rating is a 3. 

If the student is reading aloud from a text about half, the time, 
a 3 would be recorded. 

. .4 

12. Silent Practice on Outcome Behavior 

5 - Student reads silently from context nearly all the time (85-100% 
' s or 12-15 min. of a 15 min. observation). 

1 - Student does not read silently during the observation (0% of 
the time). 

(Note: Reading in context is defined as the same as #11. The examples 
of #11 are the same for #12, with silent reading.) 
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